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SYSTEMS AND METHODS FOR THE MULTISPECTRAL 
IMAGING AND CHARACTERIZATION OF SKIN TISSUE 

This application claims the benefit of U.S. Provisional Application Nos. 
60/039,218 and 60/039,407, both of which were filed on February 28, 1997, and are 
incorporated by reference, herein. 

STATEMENT REGARDING FEDERALLY 
SPONSORED RESEARCH 

This invention was made with Government support under NIH/lsiational 

Cancer Institute Contract No. 2-R44-CA60229-02A1 and U.S.A.F. Phillips Laboratory 

Contract No. F29601-95-C-0125. The Government has certain rights in this invention. 

FIELD OF THE INVENTION 

This invention relates to methods and systems for the computer controlled 
image analysis of digital skin tissue at a plurality of wavelengths, which may include 
those outside of the red-green-blue bands. The methods and systems further include 
the automatic characterization of the condition of the skin tissue, based on 
automatically computed values of parameters which are functions of characteristics of 
the skin tissue, based on the digital images. Skin lesions can be analyzed for 
detemnining whether the lesion is a melanoma, for example. Systems for digitally 
imaging and analyzing skin tissue are disclosed, as well. 

BACKGROUND OF THE INVENTION 

Melanoma is a usually fatal skin cancer, unless it is detected and 

surgically removed in its earliest stages. Early detection of malignant melanoma is 
difficult because early melanomas, those having a Breslow thickness less than 1 mm. 



share many diagnostic features with benign lesions, such as dysplastic nevi or atypical 
nnelanocytic nevi. 

To aid in the analysis of lesions, conventional photography, referred to as 
"clinical Imaging", has been used to Image the lesion for further study. The 
effectiveness of clinical imaging can be compromised, however, by specular reflection 
by the skin. Polarizers have been used for polarized imaging, which minimizes 
specular reflection. 

Demnoscopy is another technique for examining skin, in which specular 
reflection is minimized. Dermoscopy also assists In clinically differentiating melanoma 
from its benign simulants by enabling the observation of features of pigmented 
melanocytic lesions that are not discernible by the naked eye. In dermoscopy, the skin 
is made more transparent to light by providing an oil layer over the skin, in front of the 
optical system. A glass plate is placed over the oil layer. The oil has an index of 
refraction between the index of refraction of the horny layer of the skin and the glass 
plate. Standard magnifying optics may be used to enlarge the structures rendered 
visible on and under the surface of the skin by the oil layer. The region of interest can 
then be examined visually. Slides of the region of interest can be made, as well, for 
future study. 

Despite their similarities, most malignant melanomas differ in certain of 
their characteristics from other melanocytic lesions. A major advance in characterizing 
skin lesions based on certain of the observable differences between malignant and 
other lesions is the "ABCD" rule, where A=asymmetry, B^border irregularity, C=scolor 
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variability, and D=diameter greater than 6 mm. A corresponding ABCD rule, where "D" 
refers to dermoscopic structures, such as brown globules, black dots or pigment 
networks within the lesion, is applied to dermoscopic images. Because the clinical and 
dermoscopic applications of these rules are subjective, they are not very reliable. 

When skin is illuminated by light, the light can be re-emitted by reflection, 
scattering or fluorescence. It is known in the art that the re-emission of light absorbed 
at different wavelengths by a region of interest of skin can provide different information. 
For example, as the wavelength of the light increases, its depth of penetration into the 
skin or other tissue also increases. Chromophores at different depths in the tissue 
therefore absorb and re-emit light at various wavelengths. Melanin and hemoglobin are 
examples of such chromophores. 

Since the unaided eye cannot perceive light outside of the visible region 
or low-contrast structure in visible-light images, information which may be useful in 
diagnosing a lesion may not be directly observable. Digital acquisition and processing 
of dermoscopic images may, therefore, improve diagnostic reliability by employing 
more of the infonmation residing in such images that is not directly observable. There 
have therefore been attempts to use objective, computer-based, image analysis 
algorithms that can discern meaningful differences between benign and malignant 
melanocytic lesions with sufficient accuracy. 

Computer processing of images requires that the image be in digital form. 
A digital image is an array of digital signals whose values are a function of certain 
characteristics of the subject of the image. When imaging skin lesions, the 
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digital images comprise digital signals whose values are a function of the re-emission 
characteristics of the skin and lesion, at different spectral bands of light. The array is 
obtained by spatial sampling and quantizing the intensity of images obtained with film 
or directly by electronic cameras. Practical limitations on the number of picture 
elements or pixels per unit area of image detemiine the achievable spatial resolution of 
the digital image. The digital image typically needs to be segmented to separate the 
digital signals which are a function of the skin lesion from the digital signals which are a 
function of the surrounding skin. 

Coniputer aided analysis has also been used to classify skin lesions 
using quantitative values indicative of particular characteristics of lesions, referred to 
as parameters. Based on histopathological diagnosis of lesions, algorithms have been 
developed which use linear or non-linear classifiers to combine parameters provided by 
the operator of an imaging device or a physician or computed by a processor, to yield a 
value which can be used to classify the lesion. Because some of the steps in the 
computer-aided analysis of which we are aware depend on subjective judgments of an 
individual, such analyses may provide highly variable results. 

The images heretofore available have been obtained with commercially 
available red-green-blue color imaging apparatus. Color photographic transparencies 
of skin lesions have been digitized and skin lesions have been directly imaged with 
"three-chip" digitizing cameras. Such cameras employ broad-band filter bandpasses 
that are ultimately based on the wavelength response of the human visual system and 
have large regions of overlap. 
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Electronic images may also be obtained in narrower, non-overlapping 
filter bandpasses, which may reveal additional, wavelength-dependent differences 
between the images of melanomas and of benign lesions. However, such devices have 
had poor resolution and/or poor signal-to-noise characteristics which prevent the 
acquisition of digital images of melanocytic skin lesions of sufficient quality for effective 
application of machine vision techniques for lesion diagnosis. 

Existing imaging systems and processes also tend to suffer from an 
inability to provide the required repeatability of the values of extracted lesion 
parameters, due in part to a lack of standardization with respect to spatially varying 
artifacts. The parameters, therefore, lack invariance to lighting and image exposure 
conditions, for example. Obtaining high signal-to-noise ratios in images recorded in 
narrow filter bandpasses, when exposure times are sufficiently short that the skin is 
effectively "frozen" during the exposure sequence, has also been difficult. In addition, 
since the optimum wavelengths for automatic characterization may not be the optimum 
wavelengths for visual observation, it may be difficult to reconstruct high-fidelity color 
images from the digital images for visual interpretation by a clinician. 

The assessment of wounds and burns through the appearance of color 
images present similar challenges. Existing technology for the imaging of skin in vivo 
for these purposes is also inadequate. Practical solutions to the problems of employing 
multispectral digital imaging of skin for the analysis of lesions, wounds, or other 
conditions have not been found. 



SUMMARY OF THE INVENTION 

The methods and systems of the present invention provide for the 
acquisition of digital images of skin at a plurality of spectral bands to automatically 
characterize the condition of the tissue based on the digital images. Spectral 
wavelength bands within and outside of the visible band may be used. In accordance 
with the present invention, a pigmented skin lesion can be characterized as malignant 
or benign, for example. The digital images comprise a plurality of digital signals whose 
values are functions of the condition of the tissue. The digital images acquired are 
subjected to objective and quantitative analysis by a digital processor to detect and 
identify abnormalities. The analysis includes image segmentation, parameter 
estimation and characterization of the skin. The estimation and characterization steps 
are automatic. The segmentation step may be automatic, as well. Subjective 
judgments are therefore minimized or eliminated. 

It has been found that generating the segmentation mask from a digital 
image acquired with light in a spectral band which does not penetrate deeply into the 
skin, such as a blue spectral band, provides superior results. After segmentation, 
estimated values which are functions of characteristics of the lesion, such as its texture, 
asymmetry, blotchiness, and border irregularities, are computed and used to 
automatically characterize the condition of the skin. Digital signals corresponding to 
hair or blob-like structures are preferably removed during segmentation. 

In accordance with the present invention, a method for characterizing the 
condition of a region of interest of the skin, wheirein the absorption and scattering of 
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light in different spectral bands by the region of interest is a function of the condition of 
the skin, is disclosed. The method comprises illuminating the region of interest of the 
skin by light in at least three spectral bands and digitally imaging the region of interest 
at the at least three spectral bands with the light re-emitted by the skin to generate 
digital images comprising digital signals whose values are a function of the condition of 
the skin, the digital images are provided to a processor which segments the digital 
images by generating a segmentation mask from a digital image In any one of the at 
least three spectral bands, computes at least one estimated value for each digital 
irnage at each spectral band which is a function of a characteristic of the region of 
^ interest within the segmentation mask, characterizes the condition of the skin based on 
n the estimated values, and outputs the characterization of the condition of the skin. 
=p Preferably, the segmenting, estimating and characterizing steps are conducted without 
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Q the intervention of an operator. Useful parameters include measures of the texture, 
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b{ asymmetry, blotchiness and border irregularity of the portion of the region of interest. 
m The digital images may be obtained by directly imaging the region of 

m interest with a digital camera, or digitally imaging color slides of the region of Interest, 
through appropriately filtered light. 

The characterizing step may include comparing a weighted combination 
of the parameter values against a threshold value. The weight coefficients for each 
parameter value and the threshold value may be selected based on a training set of 
images of lesions or other skin conditions, whose condition has been determined. 
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preferably through histological examination by a plurality of doctors. Preferably, 
specificity is maximized under the constraint of 100% sensitivity to melanoma. 

In accordance with another aspect of the invention, a system for 
characterizing the condition of a region of interest of skin includes means for 
illuminating the region of interest with light in at least three spectral bands and a 
camera for acquiring digital images of the region of interest based on the light re- 
emitted from the illuminated region of interest at each of the spectral bands. The digital 
image comprises digital signals whose values are a function of the condition of the 
region of interest. A digital processor segments the digital images by generating a 
segmentation mask from a digital image in any one of the at least three spectral bands, 
y and computes at least one estimated value for each digital image at each spectral band 
£ which is a function of the texture of the portion of the region of interest within the 

2 

u segmentation mask. The processor characterizes the lesion based on the estimated 
^ value or values. The other parameters discussed above may be used, as well. 
'~ The camera may be a single-chip or multiple-chip charge-coupled device 

m which detects light in a plurality of spectral bands between the near ultraviolet to near 
infrared. The filter means may be a plurality of interference filters mounted on a wheel 
for stepping any filter into a position intercepting the light from the light source. 
Preferably, at least one of the spectral bands has a center which lies between about 
400 and 500 nanometers, at least one of the spectral bands has a center which lies 
between about 500 and 600 nanometers, and at least one other spectral band has a 
center which lies between about 750 and 1000 nanometers. 
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DESCRiPHON OF THE FIGURES 

Fig. 1(a) is a schematic illustration of a method and system of imaging a 
region of interest of skin in accordance with the present invention; 

Fig. 1(b) is a schematic illustration of a plurality of narrow spectral 
bandwidths which may be used to Illuminate the skin in the embodiment of Fig. 1(a); 

Fig. 1(c) is a schematic illustration of alternative methods and systems 
for digitizing and analyzing color photographic slides of a region of interest of skin; 

Fig. 2 is a schematic illustration of preferred illumination and imaging 
portions of a computer controlled imaging system for direct imaging of a lesion; 

Fig. 3(a) is a flow chart of a calibration procedure for use with the present 

invention; 

Fig. 3(b) is a flow chart of a method of processing images for classifying 
lesions as malignant or benign, in accordance with the present Invention; 

Figs. 4(a) and 4(b) are histograms of a malignant melanoma and of an 
atypical melanocytic nevus, respectively, showing two peaks in each histogram; 

Figs. 5(a) and 5(b) are histograms of another malignant melanoma and 
another atypical melanocytic nevus, respectively, showing three or more peaks in each 
histogram; 

Figs. 6(a) and 6(d) are digital images in the blue spectral band of another 
malignant melanoma and another atypical melanocytic nevus, respectively; 

Figs. 6(b) and 6(e) are digital images of the Images of Figs. 6(a) and 6(d) 
respectively, after thresholding; 



Figs 6(c) and 6(f) are digital images of the images of Figs. 6(a) and 6(d), 
respectively, after iterative thresholding; 

Figs. 7(a) and 7(d) are digital images in the blue spectral band of another 
malignant melanoma and another atypical melanocytic nevus; 

Figs. 7(b) and 7(e) are digital images of Figs. 7(a) and 7(d). respectively, 
resulting from iterative processing and showing dark blobs outside the lesion area; 

Figs. 7(c) and 7(f) are digital image masks of Figs. 7(b) and 7(d), 
respectively, resulting from image cleaning; 

Figs. 8(a) and 8(e) are digital images in the blue spectral band of another 
malignant melanoma and another atypical melanocytic nevus, respectively, showing 
hair; 

Figs. 8(b) and 8(f) are reverse intensity contrast images of the lesions of 
Figs. 8(a) and 8(e), respectively; 

Figs. 8(c) and 8(g) are digital images resulting from an averaging process 
applied to the images of Figs. 8(a) and 8(b), to remove hair; 

Figs. 8(d) and 8(h) are binary lesion masks resulting from the 
segmentation of the images of Figs. 8(c) and 8(g), respectively; 

Fig. 9 is a spatial filter used to remove hair; 

Figs. 10(a) - 10(c) are segmented digital images in the blue, green and 
red spectral bands, of the malignant melanoma whose histogram is shown in Fig. 5(a); 
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Figs. 10(d) - 10(f) are segmented digital images in the blue, green and 



red spectral bands, of an atypical melanocytic nevus whose histogram is shown in Fig. 



Fig. 1 1 is a chart of lesion parameters and their associated diagnostic 
accuracy, sensitivity and specificity when used individually; 

Fig. 12 is a plot of linear classifier values versus lesion identification 
number, for 41 malignant melanomas and 104 atypical melanocytic nevi; 

Fig. 13 is a plot of linear classifier values versus lesion identification 
number for 24 superficial spreading melanomas and 16 melanomas in-situ; and 

Fig. 14 is a plot of lesion parameter versus Breslow thickness for 24 
superficial spreading melanomas. 

DESCRIPTION OF THE INVENTION 

Fig. 1(a) is a schematic illustration of a method and system 1 in 
accordance with the present invention, by which images of the skin 2 are acquired by a 
camera nearly simultaneously at a plurality of spectral bands, h, 1=1, 2,... M, that are 
preferably effectively non-overlapping, as shown schematically in FIG. 1(b). The skin is 
illuminated by a source of white light 3, which is filtered by narrow passband filters 4. 
The filtered light is preferably conveyed to the skin 2 through a fiberoptic illuminator 5. 
The light re-emitted by the illuminated skin is imaged by a low-noise, high-resolution 
monochrome camera 6, which is preferably an electronic charge-coupled ("CCD") 
camera. Digital images output by the camera 6 are provided to a computer 12 for 
processing. 
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The computer 12 includes a digital interface 12a, a memory 12b and a 
digital processor 12c. A display 19 is preferably provided as well. The computer 12 
includes an input to a digital interface 12a for receiving the digital images. A memory 
12b stores the digital images, and the software controlling operation of the imaging 
system, the image processing, and the classification and characterization of the lesion. 
The digital processor 12c, und6r control of the software, performs the calculations. The 
computer 12 has an output connected to a display 19, which can display the processed 
images and the results of the classification and characterization procedures for each 
image. The computer 12 also preferably has outputs connected to the source of light 3 
and the camera 6, for controlling their illumination level and exposure times, 
respectively, as described below. 

The image processing, classification or characterization and other 
programs can be implemented on a personal computer, using a programming 
language, such as FORTRAN or C. The memory 12b which stores the software can be 
any convenient media readable by the computer 12, such as the hard drive of the 
computer, read only memory, random access memory with a battery backup, electrically 
programmed ROM, electrically erasable ROM, floppy disc, or CD ROM. Other suitable 
media may be used, as well. 

When the filter bandpasses have minimal overlap, as in FIG. 1(b), each 
monochromatic image will contain spectrally independent Information. Such spectral 
separation is believed to be useful for differential diagnosis of skin lesions that contain 
varying amounts of melanin, and of hemoglobin in different oxidation states, for 
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example. Spectral separation is also believed to be useful in distinguishing granulation 
of tissue and other structural details of wounds in various stages of healing. One or 
more of the wavelength bands may lie outside the visible region, such as in the near 
infrared and/or the near ultraviolet, as long as the wavelength is within the response 
range of the combined optical system including the electronic camera 6. 

In accordance with another aspect of the invention, the digital images of 
skin lesions can be derived from color slides of the lesions obtained by clinical imaging, 
dermoscopy, or polarization imaging. Fig. 1(c) is a schematic illustration of alternative 
approaches to the acquisition and digitization of images of skin lesions from color 
slides. A photo camera 13 produces 35-mm color slides of a region of the skin 14. The 
camera 13 can be a Dermaphot ® camera from Heine, Optotechnik Gmbh & Co. AG, 
Gennany, for example. The slides are typically stored in an archive 15. The slides are 
subsequently reimaged by a monochrome camera 16, which may be a CCD camera, 
that photographs each slide as It is illuminated by white light that has passed through a 
sequence of bandpass filters 17 to create a color filtered version of the image. The 
slides can be illuminated at broad or narrow blue (B), green (G) and red (R) wavelength 
bands, respectively. The broad wavelength bands may overlap somewhat. In one 
example, the blue wavelength band was about 400 nm + 30 nm, the green wavelength 
band was about 550 nm + 30 nm, and the red wavelength band was about 700 nm + 30 
nm. 

Each of the filtered representations is recorded by the monochrome 
camera 16, which provides the resulting digital images 18 to an input of the computer 
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12. If a CCD camera is not used, the slide images could be digitized by any available 
commercial digitizer including three channels, one for red, one for green and one for 
blue, as long as the pixel size in the lesion plane after digitization is less than about 60 
micrometers 



N.J. The camera 16 has a photographic macro-lens, wherein t#/2.8 and f=100 mm. 
Preferably, the spatial resolution of the CCD camera 16 provides pixels having a size 
about 10-30 \im in the lesion plane. The CCD camera 16 from Electrim, Inc., has 753 X 
488 pixels. The spatial resolution with such a camera is approximately 21 x 24 nm at 
the lesion plane. Digital images of lesions obtained with this imaging system were 
used to classify lesions as malignant or benign, and to characterize lesions as invasive 
or non-invasive, as described further, below. The Electrim, Inc.. CCD camera has 16 
has rectangular pixels. A CCD camera with square pixels would simplify the calculating 
procedures. 



may be used to reimage the slides of the region of interest. The CCD camera 20 
provides digitized images for subsequent analysis by the computer 12. Broad 
bandpass filters, which are part of the CCD camera 20, produce a representation of the 
lesion as a set of three narrowband images. The filters are typically in accordance with 
CIE Standard Observer, wherein the bandwidths are broad. 

Fig. 2 is a schematic illustration of the illumination and imaging portions of 
a preferred computer controlled imaging system 22 in accordance with the present 
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An appropriate CCD camera 16 is available from Electrim, Inc., Princeton, 



Alternatively, a 3-chip CCD camera 20, indicated in phantom in Fig. 1c, 




invention, for imaging a region of interest of skin including a lesion. The electronic 
camera 23 may be a 10-bit monochromatic electronic CCD camera 23, such as the 
Xillix Model 1400, available from Xillix Technologies Corp., Canada. The Xillix camera 
is equipped with wide band, low distortion foreoptics, such as the XJP 1.9/0501 , 
available from Jos. Schneider Werke, Germany. The lower distortion fore optics and 
the camera minimize chromatic aberrations of the optical system over the 
"multispectral" sequence of exposures, enabling registration of images with sub-pixel 
accuracy, over the entire field of view. 

To ensure repeatability of imaging conditions and to minimize required 
intervention by the operator, it is preferred that the system be operated at a preset 
f/stop. For cameras such as the Xillix Model 1400, exposure times are preferably 
controlled by the computer 12 through an electromechanical shutter that can operate 
reliably between minimum and maximum exposure times tnin and tmax. 

The imaging system provides low-noise, high-resolution digital images at 
high data transfer rates, with low distortion imaging over the entire range of 
wavelengths covered by the collection of filters. The Xillix camera, discussed above, 
has a resolution at the skin surface of about 20 microns per pixel. 

The CCD camera 23 is preferably contained in a hand-held unit, 
represented schematically as box 24. The illuminator source 25 is a tungsten-halogen 
lamp whose intensity is controlled by a light-stabilized power supply 26 whose setting is 
automatically adjusted by the computer 12. A 150 watt lamp, such as the Phillips EJA, 
available from Phillips Electronics North America Corporation, N.Y., may be used, for 
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example. The output of the lamp 25 is white light. A narrowband filter 27 is provided 
between the source and an optical fiber 28. A plurality of narrowband filters, each one 
corresponding to a different spectral wavelength band, are mounted on a filter wheel 
29. Preferred filter bandwidths are listed in Table 1 , below. The filter wheel 29, which 
is driven by a stepping motor 29a, advances each filter to its proper position between 
the lamp 25 and the optical fiber 28, and holds each filter in position for a sufficient 
period of time. The computer 12 controls the motor 29a. More or fewer filters may be 
used. Appropriate lenses 14 are provided between the lamp 25 and the filter 27, and 
between the filter 27 and the optical fibers 28, as well. One or more fiber illuminators 
are provided for conveying the light from the source to the lesion. Two such 
illuminators 30a, 30b are shown in Fig. 2 for simplicity. Although the fiber illuminator 
illustrated is a bifurcated pair, a ring illuminator which provides more nearly uniform 
illumination at the skin surface, is preferred. An angle of illumination of about 20** is 
also preferred. A Fostec Model A0603 available from Fostec, Inc., N.Y., may be used, 
for example. 

The hand-held portion of the system 24 of Fig. 2, which includes the 
camera 23, may be mounted on a cantilevered arm (not shown) that can be locked into 
position. 

The digital signals making up each of the digital images output from the 

camera 23 are provided to the computer 12. The computer 12 conducts image 

processing procedures on the digital images to calibrate the images, and to objectively 

segment, estimate parameters, and classify the lesions based on the estimated 

parameters. Operator judgment is not required at any point in the process. 
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Control is maintained by the computer 12 over source intensity, filter 
position, and such camera settings as shutter timing, through the digital interface 12a. 
Key control parameters are empirically chosen on the basis of feedback from 
histograms of trial images. The intensity of the lamp 25 may be maintained at a stable 
value, commensurate with the 10-bit dynamic range of the camera 26, by monitoring a 
secondary light source, connected electrically in series with the primary light source 25. 
The light output from the secondary source is monitored by a light sensor that is 
optically isolated from light reflections associated with the primary source. Such 
reflections may be caused by the filters that are located on the filter wheel, or from the 
housing of the primary light source. This method provides optical feedback which is 
sensitive to changes in light intensity caused by changes in lamp lead resistance, for 
example, while it is insensitive to the variable amounts of light reflected from the filters, 
for example. By means of a closed control loop, the optical feedback from the 
secondary source is used to maintain long-term constant light output from the primary 
source. 

The apparatus of Fig. 2 can be used for either clinical imaging of the skin, 
wherein the skin is imaged directly, dermoscopic imaging, wherein a layer of oil is 
provided over the skin and a layer of glass placed over the oil layer, or polarized 
imaging, where a polarizer 31 is added to minimize specular reflection as shown in Fig. 
2. In dermoscopic imaging, the index-matching oil sufficiently reduces the specular 
reflection to avoid the need for a polarizer. 

Instead of being positioned between the light source 25 and the optical 
fiber 28, the narrow bandpass filters 27 may be placed between the skin and the CCD 
camera 23 to filter the light reflected, scattered and radiated from the skin 2. 
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The front end of the system preferably consists of a flat glass plate (not 
shown) for being placed over the skin. Light pressure is applied through the glass, 
onto the skin, throughout the imaging process. This helps to stabilize the region of 
interest against unwanted motion which could blur an image or which could lead to 
spatial misregistration between images obtained in different filter bandpasses. 

The preferred filters 27 for lesion imaging with a tungsten-halogen white 
light source 25 have the center wavelengths h and bandwidths (FWHM) listed in Table 
1, for 1=1,2,. ..,M, M=IO, wherein the bands are labeled by j=i-l « 0,1,. ..M-l. Such filters 
are available, for example, from Intor, Inc., Tucson, AZ. In each band, the exposure 
^. time is preferably selected to avoid saturation of the detector elements of the CCD 
^ camera 23, as well as to maximize the linear dynamic range over which the image data 

S , 3 

^ are recorded. These exposure times should be constrained to be within limits tnin and 
^ tmax which are related to the electromechanical design of the shutter, optical throughput 
of the camera 23, and avoidance of image blur associated with motion during the 

^ exposure sequence. Suitable values of tmin and tnax could be 10 ms and 550 ms, 

fu 

^ respectively, for example. The choice of center wavelength and FWHM for the filter 
2 channels, as well as the corresponding exposure times, should preferably also take into 
account the following considerations: 

(a) The center wavelength and FWHM for at least two channels should be 
chosen so that characteristic absorption lines can be differentiated, such as those 
associated with melanin and hemoglobin; 

(b) For a given set of center wavelengths, there are upper limits on the 
associated bandwidths if spectral independence of data in different channels is to be 
maintained, as illustrated in FIG. 1(a); 
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(c) Bandpasses should be chosen in the red, green and blue portions of 
the spectrum which enable "true-color" reconstruction of skin images that are suitable 
for visualization by clinicians; 

(d) The need for high signal-to-noise ratio in each image sets practical 
lower limits on the product of exposure time and filter bandwidth, especially at short 
wavelengths, where detector response falls off and lesion reflectance is low; and 

(e) The total time taken to acquire the images in all filter bands is 
preferably less than about three minutes, to minimize patient discomfort and possible 
motion. 

Q Based on considerations (d) and (e) above, and also taking into account 

g the varying spectral reflectances of skin of different colors, the exposure times in each 

s i ^ 

J filter channel are preferably adjustable, with settings based on the dynamic range 

J achieved on an empirical basis, with trial images. In this manner, both dynamic range 

^ and signal-to-noise ratio can be maximized for each filter channel. The preferred 

Q 

ry method is to choose t^pi by iteration, based on intensity histograms of images of the 
m skin obtained with trial exposures at each wavelength band. The histograms are 
m analyzed to determine the number of pixels at the saturation intensity level, Isat ~ 2*''^ 
(1023 for b=iO bits). The exposure time is decreased if the number of saturated pixels 
exceeds a predetenrtined amount, such as 0.01% of the total. Conversely, to maintain 
high signal-to-noise ratio, the exposure time is increased if a predetermined percentile 
in the histogram, 99.9%, for example, is reached at less than a preset threshold, such 
as 99.5% of Isat- The iteration process typically converges after two or three trials. 

The preferred exposure times at each wavelength for imaging skin of 
different colors to classify melanomas are listed in Table 1 , for the embodiment of Fig. 
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2 with 10 filters. It has been found that for the blue channel centered at 450 nm, the 
optinnal exposure time for dark skin is 273 ms, which is more than double the optimal 
107 ms exposure time for light skin. On the other hand, In the near infrared channel 
centered at 780 nm, the exposure times listed are much shorter, between 24 and 35 
ms, and vary relatively little with skin type. The optimal exposure time for dark skin in 
the deep blue channel at 430 nm is at tmax = 550 ms, due to the low skin reflectance and 
relatively low optical throughput of the system at this short wavelength. Even with an 
exposure time this long, therefore, the image is less than fully exposed. Greater 
throughput at this wavelength could be achieved, at the expense of poorer response in 
the infrared. 

In Table 1, the FWHM at 450 nm, is 100 nm, which is much broader than 
for other wavelengths. It has been found that where images are desired for visual 
analysis as well as computer processing, the broad wavelength band at 450 nm more 
closely matches the blue response of the human eye and is therefore preferred, in 
addition, the broad \^avelength band provides data at a higher signal-to-noise ratio. 

Table 1 appears below: 

Optimal Exposure Times (msec) vs. Skin Color 

Filter Center Filter 



Number 


Wavelength 


FWHM 


Very Light 


Medium 


Tan 


Dark 


(i=i-l) 


(nm) 


(nm) 


Skin 


Skin 


Skin 


Skin 


0 


430 


60 


405.2 


436.5 


484.4 


550.0 


9 


450 


100 


106.8 


124.9 


156.1 


273.3 


1 


500 


40 


56.4 


62.9 


88.7 


130.7 


2 


550 


10 


44.2 


50.4 


71.3 


92.9 
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3 


600 


10 


19.1 


24.0 


29.6 


39 2 


8 


650 


10 


74.5 


92.3 


104.9 


132.0 


4 


700 


10 


71.6 


86.0 


98.0 


114.4 


5 


780 


30 


25.8 


29.1 


34.9 


23.6 


6 


880 


50 


34.1 


38.6 


44.8 


46.1 


7 


950 


60 


161.1 


187.6 


205.8 


212.0 



Tables similar to Table 1 can be readily constructed based on 
experimental results for other applications, where other spectral bands may be better 
suited. For example, in the analysis of wound healing, the ability to distinguish 
O oxygenated from deoxygenated blood would be desirable. In addition, the wavelengths 
Q and exposure times in Table 1 reflect a balance between the best results for 

yj 

subsequent analysis of the images by a computer, and the best results for visual 
^ observation of the images. If visual observation is not necessary, other wavelength 

•3 

o bands and exposure times may be used. 

^ FIG. 3(a) describes how the systems and methods of the present 

invention provide for calibration of the recorded images. The calibration procedure 
permits 10-bit image data to be recorded over a large linear dynamic range in each 
spectral band, independent of skin type. The recorded images can also be 
standardized for diffuse spectral reflectance. Consistent nneasures of reflectance ratios 
in different spectral bands can therefore be obtained, despite variations in illumination 
pattem with wavelength, changes in the position of the illuminator, or aging of the lamp, 
for example. 
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First, the effects of dark current and "fixed pattern noise" are removed in 
Step 1. N images are recorded by the camera without illumination. Preferably 8 such 
dark images are recorded. The average of these N dark images, lo is calculated and 
stored in the computer 12. 

Second, spatial inhomogeneities in the illumination and in the response 
associated with each CCD pixel are removed in Step 2. A sequence of N' images of an 
illuminated flat, diffuse reflectance standard, such as a white Spectralon® target 
(R>99%) recorded. As above, N' is preferably 8. The N' images are recorded at each 
wavelength band. To average over local inhomogeneities in the reflectance standard, 
the target is moved continuously during the integration time and between exposures. A 
small motor, such as a reciprocating motor, may be used. The integration time and/or 
lamp intensity are adjusted by the computer 12 at each wavelength band until negligibly 
few of the pixels are at or just below an intensity level corresponding to saturation. 
These N' "flat-field" images are averaged to reduce the effect of spatial non-uniformities 
in the reflectance standard, as well as to improve the detection signal-to-noise ratio. 
The resulting averages are stored in the computer as iwi, where i=1,2,...M. 

Next, monochromatic "raw data" images of the skin, Isi, are captured by 
the camera and digitally acquired by the computer 12 within each filter passband, 
i-l,2,...M. If dermoscopic Imaging is used, where a thin layer of mineral oil is spread 
between the skin and a cover glass is fixed in position in front of the camera, each 
image of the skin preferably contains an image of a narrow strip of oil-free, diffusely 
reflecting gray material, held in place on the inside sur^ce of the cover glass, and 
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located along one edge of the field of view. The material may be cut out of a Kodak 
"18% gray" card. Demioscopic imaging is preferred for melanocytic lesions. The 
alternative clinical imaging mode is preferred for the imaging of wounds and burns 
because contact with the wound or burn by a cover glass is not desired. Although FIG. 
2 indicates a lesion present on the slcin 2, it will be readily understood that the same 
method will apply when a wound or burn is present, instead. In the clinical imaging 
mode, it is preferable to reduce specular reflections by employing the polarizer 31 , as 
indicated in FIG. 2. 

In either the dennoscopic or clinical imaging techniques, a fourth step is 
preferably provided, in which the raw data is compensated for dark current and fixed 
pattern noise and then normalized to the flat-field images. The dark-field compensation 
is performed by subtracting the stored average dark image Id both from the flat-field 
image Li and from the raw data image isi . The ratio of the results of these subtractions 
is then taken. This standardizes the dark-corrected raw data to the flat-field image, 
compensating for spatially varying illumination and pixel-to-pixel response variations. 
After the ratio is taken, the result is standardized to the maximum level, 2'*-1 which 
equals 1023 where b = 10 in a 10-bit data representation. The normalization process 
thus converts the image of the skin and the gray strip into a standardized diffuse 
reflectance map, with the result preserving a large linear recording dynamic range. In 
FIG. 3(a), the dark-field corrected and flat-field-normalized Images, also referred to as 
"flat-field-calibrated" images, are denoted as Isi. in any image, standardization to 
maximum level can be reinterpreted directly in terms of equivalent diffuse reflectance 
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on the basis of the average gray level over the image of the gray strip, <lgray strip>, 
and the measured average diffuse reflectance of the gray strip, which is approximately 
0.2 and varies in a known and repeatable manner with wavelength. 

Preferably, the average image intensity in the gray-strip region is also 
used to calculate weighting factors for combining three or more monochromatic images 
to provide "true-color" visualizations of lesion images on the computer 12 and display 
19. This is preferably accomplished In Step 5, where the user selects the spectral 
bands to be used in the color visualization. Step 5 can take place prior to the imaging 
session. Four bands are currently preferred for such visualization. Fitter bands j=3 and 
% 8, in a 3:2 ratio, for the red (R) channel, filter band l~2 for the green (G) channel, and 
y filter band j=9 for the blue (B) channel, in Table 1 . As indicated in Step 6 of FIG. 3(a), 
°P the relative weights applied to the R:G:B channels are preferably inversely proportional 
^ to <lgray strip:>i the average intensity over the portion occupied by the gray strip area in 
it each image. This procedure tends to reconstruct the hues and saturations in the 
m original scene to within accuracy limits associated with response nonlinearities of the 
a display 1 9. To minimize the effects of such nonlinearities with display monitors such as 
the Sony Model GDM-175E1 Multiscan monitor, for example, the viewer may prefer to 
adjust the maximum brightness in the image to correspond to the maximum image 
intensity level of the monitor. A linear transformation step, which can be readily 
accomplished by commercial software such as Adobe Photoshop, may be used. If the 
digital images are derived from photographic slides, as in the embodiment of Fig. 1(c), 
steps 5 and 6 are not necessary. 
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As indicated by daslied lines in FIG. 3(a), either the nonnalized 
monochromatic Images resulting from Step 4 or the color visualization provided from 
Step 6 can be displayed on the display 19. Any or all of the monochromatic raw 
images could be displayed as well. 

Fig. 3(b) is a flow chart of a preferred method of processing Images 
according to the present invention for characterizing the condition of a region of interest 
of the skin of a subject which includes a skin lesion. A skin lesion is selected at Step 
50. Digital images of the lesions illuminated by light filtered at the desired wavelengths 
of A,i -Xa..., are digitally recorded in Steps 52, 54, 56 and 57..., as described above. 
Each of these digital images is processed separately. In Step 58, the image taken at 
the lowest wavelength band is used to create a mask for segmentation. At Steps 60, 
62, 64, 65..., each of the images of the lesion that correspond to different wavelengths 
are segmented by means of the segmentation mask obtained at Step 58. Lesion 
parameters are computed from each of the segmented images, in Step 66. Lesion 
parameters found to be useful for classifying and characterizing the lesion and 
statistical methods for computing the estimated values of the parameters, are 
discussed further, below. The estimated values of the parameters are provided to a 
linear classifier in Step 68. The linear classifier employs a linearly weighted sum of the 
individual parameters to derive a value used to classify the lesion as malignant or 
benign. A non-linear classifier such as a Gaussian quadratic classifier or an artificial 
neural-net classifier, each employing a suitable defined merit function, may be used as 
well. In either case, the numerical value produced by the classifier is subjected to a 
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threshold test at Step 100, such that if the test is passed, the lesion is suspected to be 
malignant melanoma. If the test is failed, the lesion is declared not to be melanoma. 
The lesion could also be characterized as invasive or non-invasive with a different 
classifier. 
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I. SEGMENTATION 

The segmentation algorithms will now be described. The function of the segmentation 
algorithms is to discriminate between the lesion and normal skin in the field-of-view of the 
imaging device. This is a complex function since not only is the lesion appearance highly 
variable but so is the appearance of healthy skin due, for example, to the presence of 
blotches, hair, wrinkles, etc. The automatic algorithm described here is based on the 
images in the blue spectral band, from about 400 nanometers (nm) to 500 nm. This 
spectral band was selected because melanin absorption increases rapidly as the wavelength 
decreases. While the use of ultraviolet radiation could be advantageous, since ultraviolet 
radiation is carcinogenic, only low doses can be used. 

Segmentation in blue consists of several automatic steps: 

O Location of major peaks in the histogram 

O First, the histogram of intensity levels in the whole image is determined. Then, given 

^ a sliding window with the range of (2/^ + 1) intensity levels, the number of peaks Np in 
J the histogram over that range is determined. If iV, < 2, the range is decreased by two 
levels and if Np > 3, the range is increased by two levels and the process is repeated tmtil 

O iVp = 2 or 3. For most of the images in the data base used in this study, there are two 

i.y 

y major peaks in the histogram. Examples of such histograms are shown in Fig. 4(a) for 
^ a malignant melanoma and in Fig. 4(b) for an atypical melanocytic nevus. The lesions 
^ correspond to the lower intensity peak, since it is darker than the surrounding skin due to 
strong absorption by melanin at 400 nm. However, some lesbns are quite inhomogeneous, 
and the automatic procedure described can find 3 major peaks, as illustrated in Figs. 5(a) 
and 5(b). 

Location of the intensity threshold 

If two major peaks are found in the intensity histogram, then the threshold value It^ 
is selected to be at the histogram minimum between these two peaks, as indicated in Figs. 
4(a) and 4(b). In the case of three peaks, it has been found that, if the middle peak is closer 
to the lowest intensity peak, the threshold value is at the minimum between the middle 





and the highest intensity peak* If the middle peak is closer to the highest intensity peak, 
then the threshold value is at the miniTmiTn between the middle and the lowest intensity 
peak, as shown in Figs. 5(a) and 5(b). 

Iterative thresholding of the image 

The next step in image segmentation is iterative thresholding of the images. Given the 
intensity threshold value, image thresholding has been typically accomplished as follows. 
The intensity /(x,y) of a pixel at location {x,y) is set to zero if it exceeds Ith, i.e., 

Figs. 6(a) and 6(d) are examples of digital images of malignant melanoma and atypical 
melanocytic nevus in the blue spectral band, respectively. Figs. 6(b) and 6(e) are images 
resulting from the direct thresholding as in Eq. (1). As shown in Figs. 6(b) and 6(e), 
"holes" can appear within the lesion. Therefore, an iterative approach is preferably used. 
First, the intensity of pixels at the image edges is set to zero. Then as each iteration 
proceeds, the intensity /(a;,y) of a pixel at location (x,y) is set to zero if it exceeds Its 
and at least one of its nearest neighbors has zero intensity, i.e., 

«-) = {kv). oJ:.^,-''* = (^) 

where 

Nnn = min[/(a: - 1, y), /(x + 1, y), /(x, y - 1), /(x, y + 1), ] . (3) 

This procediu^e is iterated imtil there are no pixels with J(x, y) > If^ and a nearest neighbor 
with zero intensity. Typically, only a few iterations are required to complete this step. The 
resulting images are shown in Figs. 6(c) and 6(f). 

Figs. 7(a) and 7(d) are other examples of digital images of malignant melanoma and 
atypical melanocytic nevus, respectively. Figs. 7(b) and 7(e) are images resvdting from the 
iterative thresholding described above. Various dark blobs are seen in the images outside 
of the lesion area. These are removed in the following step. 

7-? 
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Image cleaning 

Some of the blobs in the thresholded images arise natiirally due either to dark spots on 
the normal skin or to hair as in Fig. 7(b). Others are artifacts such as the fihn edge at the 
top of the nevus image in Fig. 7(e), or dark bands at the image edges from the slide mounts. 
These bands are removed by automatically testing for their presence and then setting the 
intensity of appropriate pixels to zero. The remaining blobs could also be removed by 
determining the overall number and size, i.e., number of pixels, of connected blobs, and 
then setting to zero the intensity of pixels belonging to the small ones. However, since the 
size of some lesions exceeds 100,000 pixels, this would be computationally very intensive. 
Therefore, in practice, this step is preferably carried out as follows. First, perimeter pixels 
for all blobs in the image are located. The number of such pixels is typically less than 
10,000. Then, each of these perimeter pixels is assigned to a unique blob and its size, the 
number of perimeter pixels in the blob, is determined. The intensities of pixels belonging 
to blobs of size less than 30% of the maximmn size for that image are set to zero. This 
process is iterated vmtil all the small blobs are removed. Typically less than 10 iterations 
are needed. The intensity of all the nonzero pixels is then set to 1. The resulting binary 
lesion mask has the following property: 

Ib(x, y) = I ^ ' ^^^^^ f ) belongs to lesion ; , 
' \ 0 , otherwise. ' 

Figs. 7c) and 7(f) illustrate the resulting lesion masks. 

In the images illustrated in Figs. 7(a) and 7(d), dark hairs were either absent or were 
not adjacent to the lesion. However, there are many images with prominent dark hair 
overlapping lesions. Segmentation of such images is described in the following section. 

Segmentation of images in presence of hair 

Figs. 8(a) and 8(e) are examples of lesion images with hair. Since the segmentation 
algorithm described in the previous section would leave some of these dark hairs connected 
to the lesion, images with hair require special preprocessing to allow for hair removal from 
the normal skin. Since hair is a problem because of its high contrast with respect to the 
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normal skin in the blue, a spatial filter was designed to locate hairs. This filter, shown 
in Fig. 9, is magnification dependent. It is applied to every pixel of the original image 
sad the result is thresholded at the 5% of maximum value in the whole filtered image. 
The filtered images are shown in Figs. 8(b) and 8(f) in reverse intensity contrast, wherein 
bright features are dark. Hairs are clearly located in the filtered images. It should be noted 
that the lesion interior is almost entirely blank, indicating poor contrast between hair and 
lesion. 

Hairs are removed by an averaging process. For every non-zero pixel at (x,y) in the 
filtered image one finds the locations of 4 nearest pixels (xi,y), (x^,!/), (ar,y/), (x,y„) 
(where xi < x < Xu and yi < y < Xfu) with zero intensity. Then the intensity of every 
pixel in the original image that has non-zero intensity in the filtered image is replaced as 
follows: 

1 ^ 

^n(a:, y) = — Y^[I{xu + k,y) + I{xi - A, y) -h /(x, yu + k) + /(x, y/ - A:)] . (5) 

The images averaged in this way are shown in Figs. 8(c) and 8(g). It is seen that the 
contrast between hairs and normal skin is considerably reduced in these images. After this 
preprocessing, the segmentation algorithm described in the previous section is applied to 
the averaged image. The final binary lesion masks are shown in Figs. 8(d) and 8(h). 

The preprocessing step described above may be used for all lesion images, regardless 
of the presence of hair, enabUng fully automated lesion segmentation. However, since this 
requires more computation and causes some border blurring, the need for preprocessing 
due to the presence of dark hair is preferably indicated interactively by an operator, and 
images preprocessed only when necessary. 

Segmentation of images in other spectral bands 

Since melanin absorption is strongest in the shortest-wavelength band, the lesion 
area, which appears as a dark region in the image, appears largest in the blue spectral 
band. Since longer wavelength radiation penetrates deeper into skin, if the thickness 
of the melanin-containing layer compensates for the weak absorption, that part of the 
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lesion will appear dark even in the red spectral band. For thick melanomas, with Breslow 

thickness greater than 1 mm, one expects dark lesions even in the infrared bands. This 

was observed, for example, by Marchesini ei ai, Photochemistry & Photobiology, "In vivo 

spectrophotometric evaluation of neoplastic and non-neoplastic skin pigmented lesions. III. 

CCD camera-based reflectance imaging," Vol. 62, 1995, pp. 151-154. However, for early 

malignant melanoma, with Breslow thickness less than 1 nmi, great variability of images 

in the red spectral band has been foimd. There may be so Uttle contrast between the lesion 

and the normal skin that direct segmentation is not possible. Therefore, segmentation of 

lesion images in all spectral bands with wavelength A uses the binary lesion mask of Eq. 

(4), obtained in the shortest-wavelength band, here blue, i.e., 

/l(x, y; A) = /(x, y; A) x Ib{x, y) . (6) 

Figs, 10(a) - 10(f) are a series of images of the lesions, with their corresponding 
histograms shown in Figs. 5(a) and 5(b), segmented in the blue, green, and red spectral 
bands, as indicated. The automaticedly determined lesion borders are superimposed on 
the original lesion images. The area of dark regions is largest in the blue, 

II. LESION PARAMETER ESTIMATION 

Objective and automatic lesion classification requires quantitative algorithms for le- 
sion parameter estimation from their segmented images. Such parameters should be di- 
mensionless, independent of lesion location emd orientation in the image, and of the overall 
image brightness. It is convenient to separate the parameters used here into four broad 
classes: asymmetry, blotchiness, border, and texture. Parameters with the highest diag- 
nostic accuracy for malignant meletnoma are listed in Fig. 11, together with the vzJues of 
diagnostic accuracy, sensitivity, and specificity, for a training set of images of 41 malignant 
melanomas and 104 atypical melanocytic nevi obtained with the imaging system described 
above, with respect to Fig. 1(a) wherein the monochrome camera 16 was used to digitize 
slides. The subscript r, g, or b refers to the red, green, or blue spectral band in which the 
parameter is evaluated. K additional spectral bands are used, then each of the parameters 
coxild be computed at the additional spectral bands, as well. 
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Specific algorithms for these parameters are described below. For simplicity it is 
assxmied that the image pixels are square but the algorithms described below may be 
implemented for rectemgular pixels as well. 

Lesion Asymmetry 

Asymmetry parameter 

The lesion asymmetry parameter is based on moments of the intensity distribution. 
First, the lesion orientation angle 0 is used to locate the principal axes, which are just the 
synmietry axes for symmetric lesions. The angle 0 is computed from 

tan 26 = 2 < (x - Xc)(y - yc) > 

<(x-Xc)2>-<(y-y,)2>' 

where the lesion intensity centroid is at 

Xc = < X > and yc = < y > . (8) 

The angular brackets in Eqs. (7) and (8) denote an intensity moment, which for any 
function /(x,y) of position in the image can be computed as follows: 

where lL{x,y) is the segmented lesion image. In order to compare properties of different 
lesions, the parameters used are independent of the orientation of the lesion in the image. 
Therefore, the lesion asymmetry is determined with respect to the principal axes. The 
measure of asymmetry described here requires rotation of the image by an angle 6 so that 
principal axes are parallel to the image axes. In this principal-axis coordinate system the 
following asymmetry factors are defined: 

. En \h{xc + n, y) - Il{xc - n, y)| 

Ax = ^ ^ r / c , (10a) 

- E,E„/L(x,y) • ^'''^ 



3^ 



33 

The asymmetry parameter, 

A^A^ + Ay, (11) 

is a measure of asymmetry in the geometric shape of a lesion as well as in the distribution 
of lesion pigmentation. Asymmetry parameters tend to be larger for malignant melanomas 
than for atypical melanocytic nevi. 

Binary asymmetry parameter 

If the intensity distribution II in Eqs. (10a) and (10b) is replaced by the bmary 
intensity distribution of Eq. (4), then the corresponding asymmetry parameter Ai,in is 
the fraction of the lesion pixels which do not have a coimterpart on the other side of the 
principal axis. Thus, when based on the binary intensity distribution, parameter A^in is a 
measure of the asynmietry of the geometric shape of the lesion. 

Lesion Blotchiness 

Visuedly, many early malignant melanomas appear blotchy. In multispectral images 
there may be darker and lighter regions or blotches of rather homogeneous intensity. In 
color images, in contrast, there may be regions of different colors. Therefore, it is of interest 
to quantify such blotchiness in order to differentiate malignant from benign lesions, 

Blotchiness Parameter Based on Spatial Intensity Distribution 

The lesion is divided into Nt "topographic" regions. If Imax and are the maximum 
and minimtun intensities in the lesion in some spectral band, respectively, then a pixel at 
(x,t/) belongs to the nth region if 

For nth topographic region defined in Eq. (12), a distribution of distances of pixels in that 
region from the intensity centroid of the binary lesion mask 

^n(a;, y) = \/(x„ - + (y„ - (13) 
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is obtained and its mean value < dn> and variance Var(dn) are computed. The measure 
of lesion blotchiness based on spatial intensity distribution is 

-=£|p«. (u) 



This parameter c€ui be evaluated in every spectral band. 
Blotchiness Parameter Based on Centroids 

The lesion is again divided into Nt "topographic" regions as defined in Eq. (12). An 
intensity centroid {xc{n)^yc{n))y defined in Eqs, (8) and (9), is then computed for each 
such region separately. The blotchiness parameter based on the centroid is defined as 

where, for example, Xmax is the maximum value of Xc(n), and Ai is the lesion area in 
pixels. This blotchiness parameter is ailso determined in each spectral band separately. 

Blotchiness Parameter Based on Spatial Color Distribution 

The "color" in this analysis is not related to the visual perception of color. It is 
a quantitative descriptor of the relative intensities in red, blue, and green channels in a 
particular pixel. 

All the other lesion parameters described here involve analysis of images in each 
spectral band separately. Therefore, absolute calibration of image intensities was not 
necessary. However, in order to describe the color distribution, normalization of intensities 
in red, green, and blue spectral bands is needed, so that intensities in the three channels 
are equal for white. In the spherical color coordinate system, 



i2(x,y) = 



Ir{^, y) + Ib{x, y) + /g(x, y) ' 



"-^^^ - /h(x, y) + Jb(x, y) + Ia{x, y) ' ^^^^ 
where the subscripts i2, G, B refer to red, green, and blue spectral bands, are chosen as 
the independent variables. The lesion is then divided into color regions as follows. First 
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-R(x,y) and G(x,y) axe divided into Nr and No topographic regions. A color region is 
defined as a particular combination of two topographic regions. The total number of color 
regions is 

Nc^NrxNg . (17) 
The blotchiuess peu-ameter based on color is defined in analogy with Eq. (14); 

ESl <dn> 

Lesion Border 



C^i^LV^, (18) 



Border Irregularity Parameter 

Border irregularity is a well-known feature of malignant melanomas. It is t)rpically 
defined as the ratio of the measured lesion perimeter to the perimeter of a circle with 
the same area as the lesion. Since perimeter is difficult to estimate reliably, a statistical 
descriptor of border irregularity is used here. In addition, many lesions are elongated amd 
an eUipse is a better approximation for such lesions with regul^ borders than a circle. 

Using the binary lesion mask of Eq. (4), the lesion intensity centroid from Eq. (8), 
orientation angle from Eq. (7), area, and the aspect ratio defined as. 



AR = — = , (19) 

where primes refer to the coordinate system defined by the lesion principed axes, are 
determined. These values are then used to construct an eUipse that is the best regular 
approximation to the lesion border. For each lesion border pixel at (x6,y6), its angle with 
respect to the horizontal aixis: 

^ = t^-i(lLZlA^ (20) 

and the location of the ellipse border for the same angle (xe(<l>),ye(<f>)) are determined. 
The distribution of distances between the eUipse border and lesion border: 

dcbixb,yb)=di{xb,yb)-de{<l>), (21) 
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where 

M^b ,yb) = y/(xb - Xcf + (y6 -Vcf (22) 

and 



dM) = \/^f+yF, (23) 

is obtained and the border irregularity parameter is defined as 
Border Gradient Parameter 

Another parameter that quantitatively characterizes lesion border is the measxire of 
intensity gradients across the lesion borders over the length scale defined by n^, in imits 
of pixels. For each lesion border pixel at {x\,^yi) one determines whether pixels at (x6 ± 



fly , y6 ± n^) are at the border. If they are not, then the gradient is defined as 



n 

ts=cr 

W 1 

m G{xb,yb) = r [|/(x + n„ y) - /(a: - ng,y)\ + |/(x, y + n,) - /(x, y - n,)|] ; (25a) 

5 fa 

g otherwise, if pixels at {x ±ng,y) are not on the border, 

H <5(a^6>y6) = |/(x + n^,y)-/(x-n^,y)|, (255) 

yl or, if pixels at (x, y ± n^) are not on the border, 

^ G(x6, y6) = |/(x, y + rig)- I{x, y - ng)\ . (25c) 
The border gradient parameter is defined as 




^- <G> ' (26) 



Lesion Texture 

The description of lesion texture is particularly viilnerable to subjective judgement. 
The quantitative evaluation of lesion texture parameters is possible only using computer- 
based image analysis. While many such parameters are possible, those foimd to be helpful 
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in discriminating between malignant melanomas and atypical melanocytic nevi are de- 
scribed below. 

Texture Parameters Based on Local Intensity Variations 

Textin-e parameters are defined over a length scale Ut in tmits of pixels. For example, 
consider a pixel located at in the lesion. LfCt // and be the nniniTTiiiTn and the 
meiximum intensities in an image in the 2nt + 1 x 2nt + 1 window axoimd this pixel, i.e., 
in the range [x - n^, x + rit] and [y — rit, y + n<]. Consider a variable 

C,(x,v)=hLZA. (27) 



The first two texture parameters axe defined as: 



and 

T2 = VVar(Ci) (29) 



(30) 




Another texttire parameter uses the following variable: 

Cz{x, y) = 4w{0, 0) + w{-nt, 0) + w{nt, 0) + w{0, -n<) + w{0, rit) 
- 2[w{-nt, -nt) + w{-nt,nt) + w{nt, -rit) + rit)] 
where 

w{i,j) = /(x + :,y + ;)//(x,y). (31) 
If the value of C3 is negative, it is set to zero and the corresponding texture parameter is 



T3 = «). (3, 

Another variable that leads to a texture parameter useful for classification of melanocytic 
lesions is: 

C4(x, y) = 8w{0y 0) - u?(-nt , 0) - w{nu 0) - u;(0, -n,) - w{0, rit) 

(33) 

- w{-nt, -Tit) - w{-nt, nt) - w{nt, -nt) - w{nt,nt) 
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Again, if the value of the variable is negative it is set to zero and the corresponding texture 
parameter is 

Texture Parameters Based on Pigmented Network 

Texture parameters have also been developed by considering the properties of a pig- 
mented network. These texture parameters are measiu'es of variabihty in the area of the 
dermal papillae and in the aspect ratio (length/ width) of the rete ridges. 

Since dermal papillae appear as the brighter part of the network, one seeks all the 
local maxima over a 2nt + 1 x 2nt + 1 window. Starting from such a maximmn at (xm> Vm)? 
one finds local one- dimensional minima in eight directions (2 vertical, 2 horizonteJ, and 
4 diagonal) and locates the vertices of an octagonal region one pixel closer to the max- 
imxma intensity pixel than the minimum pixel. Such octagonal regions approximate the 
areas of dermal papillae Adp which are computed from the known location of vertices; the 
corresponding texture parameter is 



Some of the areas determined by this algorithm are due to bubbles visible in some of these 
dermoscopic images. However, since there are typically on the order of himdreds of areas, 
and on the order of tens of bubbles, the statistical pzurameters should not be significantly 
biased by this artifact. 

The aspect ratio of rete ridges is determined in a similar fashion, although one starts 
with local minima since rete ridges appezir dark in the images. The vertices of an octagonal 
region are determined in this case from one-dimensional maxima in the eight directions. 
The maocimum and minimiim extents of this region are then determined and the aspect 
ratio R is computed. This texture parameter then is 
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III* LESION CLASSIFICATION 



Selection of lesion parameters for classification was done by determining the maximum 



sensitivity to malignant melanoma and specificity for the selected twenty two parameters 
are shown in Fig. IL These parameters were then used as input to the Unear classifier. 
NonUnear classifiers may be used as well. 
For each lesion k the linear classifier is 

m = f^tVnXn{k), (37) 
n=l 

where Xn{k) are the paxameteps for the kth lesion and weights u;„ are to be determined so 
that a specified function F(L) attains maximum value. The following functions F( L) were 
used: 1) specificity under constraint of 100% sensitivity to malignant melanoma for the 
training set which included 41 malignant melanomas and 104 atypical melanocytic nevi; 
(2) classification accuracy for the 24 invasive and 16 noninvasive malignant melanomas of 
the training set; and 3) correlation with the Breslow thickness for the 24 invasive malignant 
melanomas. 

Given any training set of lesion images and corresponding set of lesion image param- 
eters, the weights that maximize F{L) are foxmd as follows. First, an initial range and 
resolution A for u;„ are selected. For each allowed set of values of u;„, the values L{h) 
are computed for each lesion. The value F{L) is determmed based on the input bom 
histopathological evaluation of the lesion based on a biopsy, such as the diagnosis of the 
lesion as benign or malignant, and the Breslow thickness for a malignant melanoma. The 
range of u;„'s is adjusted until the maximum value of F{L) is inside the range. Then the 
resolution A is reduced by a factor of two, and the process is repeated until A reaches 
specified minimum value Am.n. This procedure determines the weights u>„ only up to a 
multiplicative constant. It is noted that the classifiers resulting fi-om a particular training 
set are applicable only to images with a specific spatial and spectral resolution, and that 
lesion images obtained with a different imaging system may require the development of 



diagnostic acciuracy for malignant melanoma for each parameter computed in every spectral 
band available for the training set of images. As mentioned above, diagnostic accuracy. 
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different classifiers, by the procedures described above. 

Since detection of melanoma in its early stage significantly improves prognosis, there 
is a need for reliable methods of early detection. Clinical evaluation of melanocytic lesions 
is, however, a problem since reliable differentiation between early malignant melanoma 
with Breslow thickness less than 1 mm and atypical melanocytic nevus is difficult even 
for experienced dermatologists. In order to detect as many early melanomas as possible, 
weights in the linear classifier are preferably chosen to maximize specificity tmder the 
constraint of 100% sensitivity to malignant melanoma for the training set. For each set 
of weights, one finds the threshold value Lth of the linear classifier such that a kth lesion 
is classified as suspicious of malignancy if L(k) > Lth, and as benign otherwise. The 
resiilting classifier for the training set is 

Li = 0.025 Atin + 0.090 + 0.069 Ag + 0.160 Ar + 0.128 Ch 

+ 0.095 C/ + 0.038 B + 0.107 Tl, + 0.064T2g + 0.018 T2r (38) 
+ 0.111 r36 + 0.167 T3g + 0.268 r56 

where the weights are normalized so that the threshold value equals one. This classifier 
with sensitivity to malignant melanoma of 100% and specificity of 85% is shown in Fig. 
12. Statistical signific€ince of the specificity and sensitivity was assessed by considering 
the binomial probabilities for the value of Li to exceed the threshold for the 41 malignant 
melanomas and 104 atypical melanocytic nevi of the training set separately. At the 95% 
confidence level, one finds that sensitivity is not less than 93% while specificity is not less 
than 79%. Since there are several melanomas very close to the threshold value, a practical 
classifier may use a threshold value that is less than one. It has been fotmd that this set of 
145 images is sufficient to yield statistically significant results. A greater niunber of images 
may be used, as well. 

Some of the noninvasive melanomas, called melanomas in-situ, are confined to epider- 
mis and are 100% cxurable by surgery. The invasive melanomas, i.e., superficial spreading 
melanomas in our data base, require more extensive surgery. Therefore, it is of chnical 
interest to differentiate between invasive and noninvasive melanomas and a Unear classifier 
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was trained to perform this task. This classifier, with weights chosen to majdnaize the over- 
all classification ax:curacy for the 24 superficial spreading melanomas and 16 melanomas 
in-situ of the training set is 

L2 = -1.00 A6 - Q.UBlg - 2.47 B/r - 0.4 C6 - 0.98 C/ 

(39) 

- 1.17 r2r + 0.53 r46 + 1.98 r56 + 1.58 T66 - 0.73 

where a constant was subtracted from the classifier values to obtain the threshold value of 
zero. This classifier, with overall classification accuracy of 92.5%, is shown in Fig. 13. 

Since prognosis for invasive melanomas correlates strongly with the Breslow thickness, 
a hnear function of lesion paraimeters Q was trained to maximize the Pearson correlation co- 
efficient between Q and the Breslow thickness for a set 24 superficial spreading melanomas. 
This function is 

Q = 0.955 Ag - 1.391 Ar + 2.791 Bh - 1.320 Big + 0.146 Ct 

(40) 

+ 0.267 Cr - 0.506 B + 0.202 Tl^ + 1.476 T2r - 0.485 

and is shown in Fig. 14. Even though there are only 24 superficial spreading melanomas in 
the data base, the high correlation of 0.77 is statistically very significant since p = 9 x 10"^. 

The classifiers of Eqs. (38)-(40) are applicable to the imaging system described above, 
with respect to Fig. 1(a) wherein the monochrome ceimera 16 was used to digitize slides. 

For other imaging systems, having different spatial and spectral resolution, different 
classifiers may need to be developed, based on a sufficient data base of lesion images 
obtained with that imaging system, in «iccordance with the procedures described above. 
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The segmentation, paranieter estimation and classification programs 
described in Sections l-lll, above, can be implemented on any personal computer, 
using a programming language, such as FORTRAN or C. The program can be stored 
on any convenient media readable by a computer, such as read only memory, ("ROM"), 
random access memory with a battery backup, electrically programmed ROM 
(EPROM), electrically erasable ROM (EEPROM), floppy disc, CD ROM, or a hard disc. 
Other suitable media may be used, as well. 

While the procedures of Sections l-lll were described with respect to 
digital images obtained by imaging color photographic slides of skin lesions with a 
monochrome CCD camera 16 In accordance with the system of Fig. 1c, these 
procedures are readily adaptable to the analysis of digital images of skin lesions 
acquired directly from the region of interest of the skin with a monochrome CCD 
camera 6 of Fig. la and Fig. 2. 

In the process described in Sections l-lll, above, segmentation was 
conducted in the blue wavelength band. The segmented mask in blue was then 
applied to images in the red and green wavelength bands. Where images at additional 
wavelengths are provided, segmentation is preferably first attempted at the shortest 
available spectral band where the contrast between the lesion and normal skin tends to 
be highest because of strong absorption by melanin. The segmented mask is then 
applied to the images in the remaining spectral bands. These steps are shown in Fig. 
3b. 
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In addition, where parameters are described in terms of the red, green 
and blue wavelength bands In Section i-lll, parameters can be derived at each of the 
other wavelengths used in the systems and methods above, in accordance with the 
procedures described in Section I-lll. The additional parameters can be readily used to 
develop a classifier, also by the processes described in Section I-lll. 

The references cited above are incorporated by reference, herein. 

While preferred systems and methods for practicing the present invention 
have been described above, it is understood that departures may be made from the 
systems and methods, without departing from the scope of the present invention, which 
is defined by the following claims. 
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